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ABSTRACT 


A  modeling  strategy  for  the  validation  and  analysis  of  large- 
scale  optimization  models  is  defined  and  demonstrated.  The  strategy 
is  based  on  nine  principles  of  analysis  and  eight  principles  of 
visualization  that  are  applied  in  a  user  controlled  hierarchical 
structure  which  is  customized  to  a  particular  optimization  problem. 
For  each  model  a  set  of  analytic  tools,  such  as  spreadsheets  and 
graphs,  is  structured  to  validate  and  verify  data  and  analyze  the 
model  and  its  results.  These  tools  can  be  quickly  recreated  with 
data  from  subsequent  runs  of  the  model  and  sensitivity  analysis 
conducted  and  comparisons  made.  As  a  demonstration  the  strategy  is 
applied  to  PHOENIX,  a  large-scale  U.S.  Army  helicopter  force 
planning  model.  The  strategy  incorporates  available  technology 
using  commercially  prepared  software  and  a  computer  workstation. 
The  application  of  techniques  such  as  hypertext,  data  access  and 
backward  compatibility  enhance  the  ease  of  use  and  effectiveness  of 
this  approach. 
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I .  INTRODOCTION 


Mathematical  programming  and  optimization  are  playing  an  ever- 
increasing  role  in  industry  and  government.  It  is  believed  that 
this  trend  will  continue  well  into  the  21st  century.  Optimization 
is  now  being  used  in  medicine  and  health  care,  environmental 
management,  and  securities  and  finance  in  addition  to  the  more 
traditional  operations  research  fields  of  transportation,  production 
and  energy.  Users  realize  that  even  small  improvements  in 
operations  can  sometimes  save  millions  of  dollars.  The  widespread 
availability  of  desktop  optimizers  and  powerful  computers  has 
already  and  will  continue  to  contribute  to  this  trend.  (Hirshfeld, 
1990  and  Schultz  and  Pulleyblank,  1991) 

Operations  research  has  made  significant  contributions  in  these 
fields  by  representing  a  problem  as  a  system  of  mathematical 
equations  and  solving  for  a  single  optimal  solution.  But,  as 
advances  in  scientific  computation  allow  for  increasing  complexity 
of  mathematical  models,  it  becomes  apparent  that  they  are  more 
difficult  to  analyze.  Many  of  the  models  now  represent  a  recurring 
situation  or  involve  a  multi-period  planning  horizon.  Such  models 
are  costly  and  time-consuming  to  develop,  but  this  is  offset  by 
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their  usefulness  as  planning  tools  which  can  be  used  over  and  over 
again.  Long-term  model  use  frequently  results  in  second-generation 
users,  who  are  not  familiar  with  the  original  model  development, 
being  required  to  execute  and  analyze  follow-on  runs.  Even  model 
developers  may  have  difficulty  retaining  the  intricacies  of  the 
problem  after  significant  time  lapses  between  runs. 

Historically,  models  of  multi-faceted  problems  had  to  have  been 
cut  down  to  size  and  drastically  simplified  to  bring  them  within  the 
scope  of  human  thinking  powers,  and  computing  technology.  This  is 
no  longer  a  satisfactory  method  of  coping  with  many  of  the  complex 
dynamic  systems  being  modeled  and  solved  today.  Due  to  the  rapid 
advances  in  the  storage  and  processing  capacity  of  today's  computers 
and  the  increased  prevalence  of  human  understanding  and  use  of  these 
mechanical  abilities,  it  is  no  longer  necessary  to  simplify  these 
models.  Today's  models  are  extremely  elaborate,  modeling  a 
multitude  of  intricate  situations  through  the  use  of  thousands  of 
linear  and  non-linear  constraints  and  real,  and  integer  variables. 
Larger  and  more  complex  problems  can  now  be  modeled  and  solved 
faster  and  more  efficiently  as  computers  have  the  capability  to 
handle  larger  data  bases,  more  constraints,  more  complex 
computations,  greater  storage,  and  more  detailed  displays. 
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Just  as  the  models  are  no  longer  simple,  the  decisions 


associated  with  the  problems  are  now  more  diverse.  Specifying  an 
objective  function  value  is  often  an  attempt  to  minimise  or  maximize 
some  ambiguous  measure  of  effectiveness.  Similarly,  many  of  the 
bounds  on  constraints  are  subjective  and  not  easily  defined  as  they 
are  often  measures  of  some  intangible  quality  such  as  efficiency, 
durability,  or  some  other  performance  criteria.  The  use  of  elastic 
constraints  compensates  somewhat  for  these  subjectively  assigned 
constraint  bounds.  It  allows  the  constraints  to  be  violated  but 
only  at  some  cost  internal  to  the  model . 

Due  to  the  complexity  of  problems  being  solved  through 
optimization  an  analyst  needs  to  pursue  and  explore  formulations 
and  solutions  which  most  accurately  describe  the  situation  and 
provide  not  only  feasible,  but  practical  courses  of  action. 
Technological  advances  in  the  past  decade  have  resulted  in  a  many- 
fold  increase  in  speed  and  availability  of  computer  computational 
power  and  significant  decreases  in  the  cost  to  solve  large  models. 
This  has  supported  the  analyst's  ability  to  process  data  for  several 
optimal  solutions  or  a  range  of  feasible  solutions,  enhancing  the 
traditional  approach  of  pursuing  a  single  optimal  answer  to  a 
question  or  problem.  Comparisons  of  these  multiple  optimal 
solutions  can  be  made  and  the  effects  of  changing  constraint 
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conditions  can  be  fully  explored.  As  a  necessity,  analysts  have 
become  far  more  capable  in  their  abilities  to  conduct  sensitivity 
analysis  of  the  effect  of  data  changes  on  the  solution  even  when 
thousands  of  constraints  are  involved. 

Technology,  however,  has  not  yet  reached  all  aspects  of  the 
modeling  process.  Large,  complex  models  require  voluminous  input 
and  generate  voluminous  output.  Multiple  runs  of  the  model  compound 
this  situation.  The  analysis  '"equired  to  obtain  the  best  possible 
decision  on  the  most  realistic  model  possible  must  continue  to 
become  more  sophisticated.  Today  s  analysts  are  better  able  to  take 
realistic  account  of  the  incomplete  information  and  inconsistency 
of  a  complex  and  changing  world  and  of  the  compromises  and 
approximations  that  must  be  made  to  fit  real-world  problems  into  the 
quantitative  terms  of  a  model.  As  always,  analysis  is  the  key  to 
both  the  model  and  the  solution  of  large-scale  optimization.  (Jones, 
1988,  p.  891;  Linstone,  1985,  pp.  77-84;  and  Simon  et  al  1987,  pp. 
11-15) 

The  challenge  now  is  to  combine  these  aspects  of  modeling  into 
a  total  optimization  system  which  will  meet  the  need  for  "more 
complete  and  integrated  optimization  solutions  comprised  of  model 
preparation  facilities,  analysis  tools  including  visualization,  and 
easy  access  to  enterprise-wide  data  in  addition  to  the  solver 
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capability."  (Schultz  and  Pulleyblank,  1991,  p.  21).  As  the 
problems  become  more  complex  and  increasing  technology  allows  faster 
solutions,  the  limiting  factor  in  optimization  is  the  time  and 
energy  required  of  the  analyst  to  sort  through  input  and  output  data 
to  find  and  interpret  the  solutions. 

A  naive  view  considers  optimization  to  be  a  forward  progression 
through  four  basic  stages:  problem  definition,  data  collection, 
model  formulation  and  analysis  of  the  results.  Analysts  realize 
that  there  is  much  more  to  effectively  employing  large-scale 
optimization  models  to  solve  today's  complex  problems.  There  is  a 
widely  used,  though  seldom  recognized,  interactive  optimization 
process  in  which  analysis  is  conducted  at  each  of  these  four  steps. 
As  data  is  collected,  it  must  be  verified  and  validated.  The  model 
must  be  compared  with  the  data  to  ensure  feasibility  and  both  the 
model  and  its  solution  must  be  compared  against  the  problem  to 
ensure  fidelity.  Sensitivity  analysis  is  accomplished  by  comparing 
the  results  of  multiple  runs  of  the  model. 

The  modeling  strategy  for  interactive  optimization  presented  in 
this  thesis  seeks  to  harness  available  technology  to  vastly  improve 
and  speed  this  analysis.  It  is  an  approach  to  the  interactive 
optimization  process  that  consists  of  designing  a  series  of  analytic 
tools  such  as  graphs  and  spreadsheets  customized  to  a  particular 
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application.  These  tools  can  then  be  used  by  an  analyst  on 
subsequent  runs  and  easily  extended  as  necessary  to  new  information 
and  considerations.  It  is  especially  helpful  in  the  analysis  of 
multi-year  and  recurrent  models  and  it  reduces  the  information  gap 
and  learning  curve  between  periodic  uses  and  for  second-generation 
and  subsequent  analysts.  The  strategy  can  be  applied  to  a  large 
variety  of  models.  It  is  based  on  a  set  of  analysis  and 
visualization  principles  that  are  uniquely  tailored  to  a  particular 
model  and  driven  by  that  model  and  ,  e  needs  of  the  analyst.  The 
implementation  of  this  modeling  strategy  will  reduce  the  time  and 
effort  to  validate  data  and  interpret  solutions  allowing  the  analyst 
to  better  pursue  model  development  and  analysis  to  make  better 
decisions . 

These  analysis  and  visualization  principles  are  presented  in 
Chapter  II  along  with  a  more  detailed  discussion  of  the  interactive 
optimization  process.  Background  on  human  visual  processing  and 
computer  technology  is  discussed  in  Chapter  III.  Chapter  IV  is  a 
description  of  the  user  for  which  this  strategy  is  developed. 
Chapter  V  outlines  the  PHOENIX  model,  a  U.S.  Army  helicopter  force 
planning  tool.  Chapter  VI  demonstrates  the  application  of  the 
modeling  strategy  to  PHOENIX.  Chapter  VII  describes  the  future  of 
the  PHOENIX  model.  The  conclusions  are  presented  in  Chapter  VIII. 
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II.  THE  MODELIHG  STRATEGY 


A.  INTERACTIVE  OPTIMIZATION  PROCESS 

In  order  to  develop  an  optimization  modeling  strategy  an  in- 
depth  understanding  of  the  interactive  optimization  process  is 
required.  As  conceptualized  in  Figure  1,  the  four  basic  parts  of 
the  process  are  problem  definition,  data  collection,  model 
formulation  and  analysis  of  results.  The  main  relationships  among 
these  parts  are  depicted  by  the  forward  arrows.  The  backward  arrows 
represent  the  relationships  that  are  more  subtle,  yet  crucial,  in 
the  optimization  process. 

As  data  is  collected  it  must  be  consolidated,  verified  and 
validated.  Visual  inspection  of  the  raw  data  files  or  printouts  is 
simply  not  feasible.  The  input  data  often  originates  at  several 
sources  and  needs  not  only  to  be  verified  for  accuracy,  but  also  to 
be  compared  for  consistency  among  sources.  Without  correct  input 
data,  the  results  of  the  model  are  worthless.  The  verification  and 
validation  of  input  data  needs  to  be  an  integral  part  of 
optimization.  It  must  be  done  both  as  the  data  stands  alone  and  as 
it  is  linked  together  with  the  model. 
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All  aspects  of  the  problem  that  are  critical  to  the  solution 
must  be  enumerated  and  accounted  for  in  mathematical  formulas. 
Modeling  requires  that  an  objective  function  and  constraints,  often 
qualitative  terms,  be  quantified  in  some  way.  This  presents  a 
difficulty  in  that  real  world  situations  are  complex  dynamic  systems 
which  can  seldom  be  translated  into  black  and  white  values  and  it 
is  often  impossible  to  establish  a  hierarchy  of  these  values.  If 
there  are  mutually  exclusive  requirements,  elastic  constraints  are 
introduced  to  ensure  that  the  unfulfilled  requirement  is  at  some 
cost  internal  to  the  model. 

Both  the  model  and  its  solution  must  be  checked  against  the 
actual  problem  to  ensure  fidelity,  validity  and  practicality.  If 
not  adequately  representative  of  the  situation  being  modeled, 
changes  may  need  to  be  made  in  the  data  collection,  model 
formulation,  or  even  problem  definition.  The  analyst  must  often 
explore  alternate  solutions  obtained  by  varying  the  input  parameters 
and  constraint  conditions  to  reflect  the  dynamics  of  a  complex 
problem.  Even  small  changes  made  to  enhance  the  application  of  the 
model  can  indicate  an  entirely  new  course  of  action. 

In  order  to  conduct  sensitivity  analysis  of  large-scale 
problems,  the  results  of  multiple  runs  must  be  compared.  The 
voluminous  data  produced  by  each  run  requires  the  same  in-depth 


8 


1 


analysis  as  the  original  output.  Thorough  comparisons  between 
successive  runs  highlight  the  advantages  of  one  over  another  and  aid 
in  the  selection  of  one  optimal  or  a  range  of  feasible  solutions. 

Faster  more  efficient  computers  allow  the  solution  of  these 
large-scale  problems.  Their  final  results  can  be  displayed  and 
presented  very  effectively  using  a  variety  of  sophisticated 
presentation  tools.  Technology,  however,  has  not  yet  reached  the 
relationships  in  the  interactive  optimization  process  represented 
by  the  back  arrows.  There  will  not  be  a  general  way  to  bring 
technology  to  these  relationships  since  each  model  is  an  individual 
problem  with  a  hierarchy  of  analysis  consisting  of  unique  questions 
and  decisions  to  be  made  about  the  optimization  output.  The 
modeling  strategy  developed  in  this  thesis  can  help  to  bring 
technology  to  the  interactive  optimization  process  by  defining  a  set 
of  analysis  and  visualization  principles  that  can  be  applied  in 
different  combinations  to  all  models. 

B.  PRINCIPLES  OF  THE  MODELING  STRATEGY 

The  analysis  and  visualization  principles  that  are  the  basis  of 
the  modeling  strategy  will  be  briefly  explained  in  this  section. 
Although  it  would  be  possible  to  more  fully  develop  each  of  these 
principles,  a  more  effective  way  to  present  the  strategy  is  to  apply 
it  to  a  real,  contemporary,  complex,  and  important  model.  The 
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strategy  will  be  applied  to  the  PHOENIX  model  in  Chapter  V.  Each 
principle  will  be  demonstrated,  however  the  focus  is  not  on 
individual  principles,  but  on  the  combinations,  interactions  and 
compromises  among  these  principles  as  they  are  applied  to  an  actual 
model . 

1.  Analysis  Principles 

•  (Al)  HIERARCHICAL  STRUCTURE.  Analysis  progresses  from  a  broad 
overview  of  the  objective  function  value  down  to  a  single  piece 
of  information.  The  specific  hierarchy  will  be  designed  by  the 
analyst  and  will  depend  on  the  nature  of  the  problem.  Levels 
may  be  skipped  and  crossovers  between  branches  can  occur  to 
meet  the  needs  of  the  analyst. 

•  (A2)  USER  CONTROL.  The  analyst  is  able  to  navigate  the 
hierarchical  structure  to  answer  the  questions  that  arise  as 
part  of  a  specific  analysis. 

•  (A3)  DATA  ACCESS.  At  all  points  in  the  analysis  process,  the 
analyst  must  have  easy  access  to  specific  data  values  and  the 
relationships  among  those  values. 

•  (A4)  MULTIPLE  REPRESENTATIONS.  Different  aspects  of  the  same 
data  are  extremely  valuable  in  revealing  information  about  the 
relationships  among  data.  These  varied  representations  are 
easily  constructed  and  readily  available. 

•  (A5)  SIDE-BY-SIDE  COMPARISONS.  Side-by-side  comparisons  of 
similar  information  enhance  the  analysis  process  by  decreasing 
the  amount  of  decoding  that  is  required  and  by  highlighting 
differences . 

•  (A6)  BASE  CASE.  In  model  development  and  sensitivity  analysis, 
a  base  case  that  is  believed  to  best  represent  the  problem  can 
be  used.  Results  of  other  runs  are  compared  against  the  base 


•  (A7)  CUSTOM  MODEL.  The  nature  of  the  problem  and  the  questions 
inherent  to  it  drive  the  set  of  representations  that  are  used 
for  analysis  of  that  model. 

•  (A8)  EXTENSIBLE.  Changes  made  to  a  particular  representation  or 
piece  of  information  filter  through  all  representations  that 
contain  that  information. 

•  (A9)  BACKWARD  COMPATIBLE.  Any  additional  representations  or 
changes  to  existing  representations  made  during  analysis  of  a 
run  are  automatically  added  to  any  previous  runs  of  the  model. 

2.  Visualization  Principles 

•  (VI)  REPRESENTATION  DRIVEN  GRAPHS.  The  results  of  time-series 
models  are  most  often  displayed  with  the  time  periods  on  the 
horizontal  or  x-axis.  This  fits  in  with  the  concept  of  time 
moving  forward  and  the  general  perception  of  the  forward 
direction  as  the  one  from  left  to  right.  Alternately,  a  model 
which  see)cs  to  find  the  best  combination  of  ingredients  in 
order  to  make  some  new  product  may  be  best  represented  by 
filled  bar  charts  where  the  height  of  the  bar  represents  the 
new  product.  Pie  charts  could  also  be  used  but  are  generally 
not  considered  good  analysis  tools  due  to  the  difficulties 
encountered  in  perceiving  differences  in  angles  and  slopes 
(Tufte,  1983,  p.  178  and  Cleveland  and  McGill,  1985,  p.  829) . 
Stacked  bar  charts  are  also  very  effective  in  showing  different 
combinations  of  things  that  make  a  whole. 

•  (V2)  ZOOMING.  The  ability  to  focus  or  zoom  in  on  areas  of 
interest  or  out  to  the  big  picture  is  crucial  in  the  analysis 
process.  The  focus  could  be  on  certain  time  periods,  on 
information  pertaining  to  a  particular  constraint,  or  even  on 
the  objective  function  value.  It  allows  the  analyst  to  pursue 
a  separate  train  of  thought  or  to  redirect  analysis  in  a 
certain  direction. 

•  (V3)  DATA  ACCESS.  The  analyst  must  also  have  simple,  easy 
access  to  the  data  behind  the  graphs.  Whereas  the  graph 
provides  a  way  to  view  the  information  in  relation  to  other 
information  in  the  model,  the  actual  data  is  necessary  for 
quantitative  analysis. 
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•  (V4)  APPROPRIATE  DISPLAYS.  Not  all  information  is  most 
effectively  displayed  in  graphic  form.  The  analyst  must  decide 
when  a  graph  is  needed  and  when  displaying  the  data  in  tabular 
form  is  more  informative. 

•  (V5)  SIMPLICITY.  Visualization  tools  must  be  designed  to  reveal 
information  rather  than  draw  attention  to  the  display.  They 
should  avoid  distortion  of  the  data  and  allow  the  analyst  to 
interpret  it.  (Tufte,  1983,  p.l4) 

•  (V6)  CONSISTENCY.  Graphs  that  are  consistent  in  size,  coloring 

and  labeling  with  other  graphs  which  display  similar 
information  reduce  the  amount  of  time  required  for 

interpretation . 

•  (V7)  MULTIPLES.  An  especially  effective  method  to  display  and 

highlight  both  subtle  and  extreme  differences  is  the  use  of 
small  multiples  which  are  a  series  of  graphics,  each  showing 
the  same  combination  of  variables,  indexed  by  changes  in 
another  variable.  Since  the  design  stays  constant,  the 

analyst's  attention  can  be  devoted  entirely  to  the  changes  in 
the  data  (Tufte,  1983,  p.  170). 
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III.  BACKGROUND 


A.  HXniAN  VISUAL  INFORMATION  PROCESSING 

Extensive  research  has  been  conducted  on  human  visual 
perception  and  information  processing.  The  physiology  has  been 
explored  (Marr,  1982,  pp.  4-15)  and  numerous  experiments  conducted 
(Powers  et  al,  1983  and  DeSanctis,  1984)  to  determine  the  benefits 
of  using  representation  (graphics  and  tables)  versus  description 
(text)  to  enhance  the  human's  ability  to  process,  recall,  and 
analyze  visual  information.  Under  a  variety  of  circumstances  and 
experimental  conditions,  some  studies  conclude  the  superiority  of 
graphical  presentation  over  text  or  tables  and  some  determine  that 
no  one  method  can  conclusively  be  determined  superior.  When 
evaluated.  Powers  et  al  determined  that  the  combination  of  graphical 
and  tabular  data  was  more  effective  than  either  method  used  alone 
(1983,  p,  558) .  Dependent  variables  evaluated  in  these  studies 
differed,  but  a  typical  list  comes  from  DeSanctis  (1984,  p.  468): 

•  Interpretation  accuracy; 

•  Problem  comprehension; 

•  Task  performance; 

•  Decision  quality; 


13 


•  Speed  of  comprehension; 

•  Decision  speed; 

•  Memory  for  information  (recognition  and  recall); 

•  Viewer  preference. 

Further  research  more  specifically  explores  effective 
techniques  for  presenting  quantitative  information.  In  his  books, 
Edward  Tufte  asserts  that  “well-designed  data  graphics  are  usually 
the  simplest  and  . . .  most  powerful  of  all  methods  for  analyzing  and 
communicating  statistical  information"  and  that  the  general 
principles  of  effective  design  are  universal,  not  tied  to  language 
or  culture  (1983,  pp.  8-9  and  1990,  p.  10).  "Often  the  most 
effective  way  to  describe,  explore,  and  summarize  a  set  of  numbers  - 
even  a  very  large  set  -  is  to  look  at  pictures  of  those  numbers." 
(Tufte,  1983,  p.  9)  To  communicate  complex  ideas  with  clarity, 
precision,  and  efficiency,  Tufte  lists  the  important  elements  of 
graphical  displays  (1983,  p.  13): 

•  Show  the  data; 

•  Induce  the  viewer  to  think  about  the  substance  rather  than 
about  methodology,  graphic  design,  the  technology  of  graphic 
production,  or  something  else; 

•  Avoid  distorting  what  the  data  have  to  say; 

•  Present  many  numbers  in  a  small  space; 

•  Make  large  data  sets  coherent; 
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•  Encourage  the  eye  to  compare  different  pieces  of  data; 

•  Reveal  the  data  at  several  levels  of  detail,  from  a  broad 
overview  to  the  fine  structure; 

•  Serve  a  reasonably  clear  purpose:  description,  exploration, 
tabulation,  or  decoration; 

•  Be  closely  integrated  with  the  statistical  and  verbal 
descriptions  of  a  data  set. 

Cleveland  and  McGill  (1985,  p.  828)  caution  against  graphs  that  are 

too  elaborate  or  technologically  sophisticated.  They  have  studied 

the  link  between  graphs  and  the  human  visual  system  and  conclude: 

When  a  graph  is  constructed,  quantitative  and  categorical 
information  is  encoded,  chiefly  through  position,  shape,  size, 
symbols,  and  color.  When  a  person  looks  at  a  graph,  the 
information  is  visually  decoded  by  the  person's  visual  system. 

A  graphical  method  is  successful  only  if  the  decoding  is 
effective . 

Norbert  Enrick  (1972,  p.  2)  cites  the  values  of  "well  prepared 
charts  and  graphs"  as  creating  interest,  clearly  portraying 
relationships,  saving  time,  saving  space,  providing  a  synoptic 
overview,  unearthing  hidden  factors,  and  enhancing  thought 
processes,  analysis,  ideation  and  creativity. 

In  the  field  of  operations  research,  the  primary  applications 
of  graphic  techniques  to  date  have  been  in  the  areas  of  simulation 
and  data  analysis.  While  both  are  applicable  to  optimization, 
surprisingly  little  has  been  done  to  incorporate  graphical  methods 
into  the  analysis  process  for  large-scale  optimization  models.  But 
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as  more  complex  and  increasingly  larger  optimization  models  and  more 
sophisticated  operations  research  techniques  are  developed 
"understanding  the  behavior  of  the  underlying  system,  detecting 
trends,  debugging  and  validating  the  model  become  more  challenging. " 
(Jones,  1988,  p.6)  Graphical  and  visual  techniques  and  user 
interaction  in  the  modeling  process  can  be  employed  to  assist 
significantly  in  optimization  analysis.  Incorporating  these 
techniques  more  prominently  into  operations  research  has  been  cited 
as  a  need  in  the  current  growth  of  the  field  of  operations  research, 
and  their  potential  benefits  are  inestimable.  (Jones,  1988,  p.7; 
Bell,  1985,  p.31;  Hurrion,  1986,  p.286) 

B.  REPRESBNTATION/VISUALIZATION 

With  the  increasing  volume  of  input  data  and  results  that  can 
now  be  produced  by  large-scale  optimization,  the  problem  analysts 
face  has  changed  from  making  limited  calculations  to  being  able  to 
understand  and  interpret  the  masses  of  data  produced.  Effective 
representation  is  a  key  factor  in  the  abilities  of  the  analyst  to 
validate  and  analyze  this  information.  Text  and  tables  have  long 
been  used  by  optimizers  for  conveying  the  data  and  results  of  their 
algorithms.  But  perhaps  the  most  effective  and  universally 
understood  means  of  representation  is  through  the  use  of  graphics. 
Computer  graphics  programs  that  are  used  to  help  scientists 
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visualize  and  thereby  better  understand  their  research  problems  are 
categorized  as  scientific  visualization  (Rivenbark,  1989  ) .  Along 
the  lines  of  the  old  adage  “a  picture  is  worth  a  thousand  words", 
these  visual  images  can  convey  more  information  and  reveal  aspects 
and  relationships  not  as  easily  discerned  by  analyzing  formulas  or 
perusing  numerical  values.  “Using  computer  generated  images  and 
human  vision  in  scientific  visualization  .  .  .  (can)  convey  a 
tremendous  amount  of  information  in  a  short  period  of  time," 
(Nielson,  1989,  p.  10).  The  strength  of  visualization  is  its  use 
of  the  greatest  processor  available:  the  human  brain,  according  to 
Lloyd  Treinish,  a  computer  scientist  with  NASA's  National  Space 
Science  Data  Center  (NSSDC) ,  because  it  "takes  advantage  of  the 
inherent  power  of  the  human  visual  system. "  (Rivenbark,  1989,  p, 
36) 

Craig  Mundi,  Research  and  Development  vice  president  for 
Alliant  Computer  Corporation,  Littleton,  Massachusetts,  divides 
visualization  into  two  types:  statistically  produced  data  sheets 
from  which  the  user,  or  computer,  extracts  a  static  graphical 
representation,  and  models  that  the  user  dynamically  manipulates  in 
real  time  (Jones,  1988) .  This  real-time  animation  is  becoming  more 
frequently  utilized  in  the  field  of  operations  research  for 
animation  of  algorithms  and  simulations  models.  Animation  allows 
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the  analyst  to  interact  with  and  thereby  redirect  or  change  the 
actual  problem  during  its  solution.  The  solution  or  the  algorithm 
can  also  be  animated  showing  the  user  the  actual  process  behind  the 
model  and  the  steps  involved  in  solving  it.  Currently,  however, 
technology  cannot  accommodate  the  animation  of  large-scale 
optimizations  with  their  multitude  of  variables  and  constraints. 
Three-dimensional  computing  has  improved  Operations  Research 
visualizations  greatly,  but  scientific  computing  has  yet  to  master 
graphing  the  thousand-dimensional  and  the  human  mind  would  have 
difficulty  comprehending  it. 

Optimization  does  not  now  allow  run-time  interaction  to 
redirect  the  model  or  make  changes  during  computation.  The  reason 
for  this  is  found  in  the  Simplex  procedure  involved  in  linear 
programming.  The  assortment  of  variables  and  constraints  that  make 
up  the  optimization  algorithm  form,  in  effect,  a  multi-dimensional 
polytope  whose  extreme  points  bound  the  feasible  solutions.  The 
Simplex  optimization  process  searches  these  points  for  the  optimal 
solution.  Whether  using  the  traditional  Simplex  algorithm  which 
progresses  from  one  adjacent  vertex  to  another  until  there  is  no 
improved  point,  or  the  more  recently  developed  Karmarkar  algorithm 
(Hamilton  and  Stein,  1989,  p.36)  which  employs  a  shortcut  to  work 
through  the  center  of  the  polytope  instead  of  on  the  surface,  any 
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run-time  interaction  would  change  the  composition  of  the  polytope 
and  invalidate  the  systematic  optimization  process.  The  use  of 
static  graphical  representations  of  visualization  are  more 
appropriate,  then,  to  improving  the  portrayal  of  large-scale 
optimizations . 

What  visualization  can  provide  to  large-scale  optimization  is 
enhancement  and  clarity  of  voluminous  input  and  results.  It  can  be 
used  as  a  vehicle  in  focusing  and  directing  analysis.  The  strategy 
described  here  employs  these  static  visualizations  in  an  interactive 
environment  where  analysis  and  changes  can  be  incorporated  between 
successive  runs  of  the  optimizer  to  explore  alternate  decision 
strategies.  The  development  of  graphics  programmed  directly  into 
the  output  production  excuses  the  analyst  from  having  to  become  a 
computer  programmer  and  from  performing  labor  intensive  data 
manipulations.  It  speeds  the  preparation  of  graphic  representations 
of  the  model  and  eliminates  the  need  to  "reinvent  the  wheel"  as  the 
analyst  elects  to  display  different  formats,  change  input  values  and 
parameters,  and  validate  and  make  comparisons  on  multiple  runs  of 
the  optimization  (Rivenbark,  1989) . 

These  changes  serve  to  produce  better  problem  solving  in  the 
field  of  operations  research  and  enhance  the  credibility  of  analysts 
in  their  work  with  managers  and  decision  makers  who  prefer  a  more 
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socio- technical  approach  to  problem  solving.  To  fully  realize  the 
potential  of  mathematical  optimization,  the  effectiveness  of 
algorithms  and  their  implementation  must  be  developed  to  solve  the 
actual  problems  of  the  user  (Schultz  and  Pulleyblank,  1991)  .  By 
capitalizing  on  the  speed  and  power  of  delivery  systems  in  the 
development  of  a  total  optimization  system,  the  modeling  strategy 
will  greatly  enhance  the  capabilities  and  latitude  of  the  analyst. 

C.  COMPUTER  TECHNOLOGY 

The  great  strides  made  in  computer  technology  have  resulted  in 
a  decline  in  price  coupled  with  a  rise  in  capability.  This  puts 
extremely  powerful  machines  within  easy  reach  of  analysts  and 
researchers.  These  machines  can  solve  exceptionally  large,  complex 
problems  in  relatively  short  periods  of  time  as  they  are  adequately 
equipped  with  memory  to  store  the  tremendous  amount  of  data  and  the 
intricate  relationships  among  its  elements  that  these  problems 
demand.  Along  with  this  increased  power  come  fast  enhanced  graphics 
that  can  provide  the  user  with  high  quality  visualization  products 
for  both  analysis  and  presentation.  This  is  especially  important 
for  operations  researc)i,  and  even  more  so  for  optimization,  as  it 
allows  the  modeling,  solving  and  analysis  of  production  sized 
problems  with  a  degree  of  fidelity  that  has  never  before  been 
possible,  Gregory  M.  Nielson,  in  an  article  in  Computer  magazine 
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(1989,  p.lO),  captures  the  advances  of  computer  technology  and 


highlights  the  direction  for  the  future: 

Advances  in  scientific  computation  allow  increasing  complexity 
of  mathematical  models  and  simulations.  This  results  in  a 
closer  approximation  to  reality,  which  enhances  the  possibility 
of  acquiring  new  knowledge  and  understanding  .  .  .  The  problem  is 
to  convey  all  of  this  information  to  the  scientist  to 
effectively  use  human  creativity  and  analytic  capabilities. 

Of  the  newly  affordable  computer  systems,  the  workstation  may 
have  the  most  profound  effect  on  the  operations  research  analyst. 
In  addition  to  having  the  power  and  speed  of  the  latest  technology, 
it  has  several  other  characteristics  which  enhance  the  capabilities 
of  an  analyst.  One  advantage  is  the  high  speed  architecture  which 
allows  quick  transfers  among  many  different  environments  such  as  the 
model,  the  spreadsheet  and  the  graphics.  Additionally,  most 
workstations  employ  a  windowing  system  which  provides  a  set  of 
programming  tools  and  commands  for  building  the  menus,  windows,  and 
dialogue  boxes  that  appear  on  a  screen.  These  two  features 
alleviate  the  difficulties  and  inconveniences  of  switching  computer 
environments  and  facilitate  the  simultaneous  use  of  multiple 
applications  for  comparison  and  consolidation. 

While  workstations  are  self-contained  computers,  they  are  also 
an  integral  part  of  the  networking  concept.  In  a  network, 
information  is  shared  among  all  the  systems  which  are  linked  to  each 
other  via  this  network.  In  addition  to  sharing  information. 
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networks  can  also  share  the  work  required  to  solve  large-scale 
problems.  One  computer  is  in  primary  control  of  a  problem  but  can 
distribute  solving  tasks  to  other  computers,  including  other 
workstations,  micro-computers  and  even  a  mainframe.  The 
distribution  is  based  on  both  the  capability  and  availability  of  the 
machines  on  the  network.  When  a  task  is  completed,  its  solution  or 
status  report  is  sent  back  to  the  controlling  computer  for 
consolidation  with  the  rest  of  the  problem  until  the  entire  job  is 
complete.  This  process  maximizes  the  effective  use  of  the  computers 
and  minimizes  the  time  required  to  solve  a  problem.  The 
interactive  optimization  process  discussed  in  this  thesis  does  not 
directly  rely  on  networked  problem  solution,  but  there  is  potential 
for  links  with  other  systems  to  take  full  advantage  of  a  network 
during  the  course  of  an  optimization. 
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IV.  DESCRIPTION  OF  TARGET  USER 


Currently,  off-the-shelf  software  exists  to  bring  the  methods 
of  optimization  to  the  general  user.  Several  packages,  such  as  VINO 
and  Lotus  1-2-3  utilize  spreadsheets  and,  with  simple  instructions 
for  input  of  objective  function  and  constraint  values,  will  produce 
an  optimal  solution  without  the  requirement  of  user  comprehension 
of  optimization  principles.  Perhaps  one  of  the  most  capable 
products  for  simple  linear  optimization  is  What's  Best  which 
integrates  the  flexibility  of  the  Lotus  1-2-3  spreadsheet  with  the 
power  of  LINDO  (Bodily,  1986,  pp,  41-42) .  These  programs,  while 
useful  for  more  simple  tasks,  are  not  sophisticated  enough  for  the 
complex  problems  encountered  by  operations  research  analysts. 

The  process  developed  in  this  thesis  will  address  optimization 
at  this  higher  level.  It  is  designed  for  a  sophisticated  user 
trained  in  operations  research  and  analysis.  It  requires  an  in- 
depth  understanding  of  the  theory  and  mechanics  of  mathematical 
programming,  including  the  principles  of  sensitivity  analysis.  A 
working  knowledge  of  data  analysis  techniques  with  emphasis  on 
graphical  representations  is  also  essential.  This  system  is  not 
intended  for  a  casual  user.  A  level  of  involvement  is  expected  that 


would  justify  the  construction  of  a  project-specific  application  of 
the  process  developed.  It  is  envisioned  that  the  analyst  will  have 
a  long-standing  relationship  with  a  particular  project  and  with 
optimization  projects  in  general.  This  will  ensure  that  the  process 
is  used  to  its  fullest  extent  and  that  the  user  will  derive  the 
maximum  benefit  from  it.  Additionally,  many  long-term  projects  are 
designed  for  use  as  recurring  decision  support  models.  The 
techniques,  therefore,  must  be  sufficiently  simple  and  generic  to 
be  successfully  transferred  among  qualified  users. 

As  a  minimum  it  is  expected  that  the  user  will  have: 

•  6  semester  hours  of  graduate  credit  in  linear  programming  and 
integer  programming  methods. 

•  3  semester  hours  of  graduate  credit  in  data  analysis 
techniques. 

•  familiarity  with  spreadsheets  and  their  related  graphics 
packages. 

•  experience  with  a  computer  workstation  environment. 
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V.  THE  PHOENIX  MODEL 


A .  PURPOSE 

PHOENIX  is  currently  used  by  the  U.S.  Army  Concepts  Analysis 
Agency  (CAA)  as  a  decision  aid  for  helicopter  force  planning.  It 
was  developed  in  late  1987 /early  1988  in  response  to  the  realization 
by  the  U.S.  Army  that  it  had  no  comprehensive  plan  for  modernizing 
its  helicopter  fleet.  The  helicopter  fleet  was  composed  of  mostly 
Vietnam-era  aircraft  which  were  nearing  the  end  of  their  useful 
lives.  The  Army  Aviation  Modernization  Trade-off  Requirement 
(AAMTOR)  study  (Force  Systems  Directorate,  1988)  was  commissioned 
to  develop  a  comprehensive  force  planning  decision  aid  and  the 
PHOENIX  model  was  the  result.  As  described  by  Brown  et  al  (1991), 
PHOENIX  "captured  complex  procurement  and  modernization  tasks  in  an 
optimization-based  decision  support  system  .  .  .  which  recognizes 
yearly  operating,  maintenance,  retirement,  service-life  extension, 
and  new  procurement  costs  while  enforcing  constraints  on  fleet  age, 
technology  mix,  composition  and  budgets  over  a  multi-year  planning 
horizon" .  The  final  report  of  the  AAMTOR  study  describes  the  model 
development,  data  collection  and  analysis  in  full  detail. 


B .  DESCRIPTION 


The  portions  of  the  model  which  are  critical  to  the 
understanding  of  this  thesis  are  summarized  in  this  section  and  the 
following  one.  (Force  Systems  Directorate,  1988)  .  PHOENIX  is  a 
mixed  integer  linear  program  (MILP)  variant  of  a  classic  operations 
research  optimization  problem,  the  equipment  replacement  model. 

Since  the  real  scenario  is  more  complex  than  the  classic  problem, 
the  PHOENIX  model  is  also  more  elaborate  (p.  3-1)  .  PHOENIX 
addresses  such  concerns  as  multiple  missions  of  the  Army  aviation 
fleet,  budgetary  limits,  fixed  costs  associated  with  production  and 
multiple  criteria  for  mission  fulfillment  that  are  not  accounted  for 
in  the  basic  equipment  replacement  model .  The  PHOENIX  model  was 
solved  using  a  commercial  quality  optimization  package,  the  "X- 
system"  (p.  3-10).  The  most  difficult  scenario  that  was  solved  was 
over  a  25  year  planning  horizon.  It  contained  288  binary-valued 
decision  variables,  9579  continuous  decision  variables  and  3737 
constraints  (p.  3-10) . 

C.  GOALS  AND  ASSUMPTIONS 

The  objective  of  the  PHOENIX  model  is  to  minimize  the  sum  of 
the  operations  and  maintenance  annual  expenditures  and  the  penalties  * 

associated  with  constraint  violations.  The  annual  budget  and 
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mission  fulfillment  requirements  are  included  in  the  constraints 
of  the  model.  The  model  determines  (p.  5.1): 

•  When  (time  period)  aircraft  production  lines  begin  and  end 
production,  or  if  they  do  at  all. 

•  How  many  aircraft  are  purchased  from  each  production  line  in 
each  time  period. 

•  How  many  operational  aircraft  in  a  cohort  are  improved  through 
a  Service  Life  Extension  Program  (SLEP)  in  each  time  period. 

•  How  many  operational  aircraft  in  a  cohort  are  retired  in  each 
time  period. 

Some  of  the  )cey  assumptions  of  the  model  are  (p.1-8): 

•  Aircraft  are  purchased  and  supported  as  cohorts  composed  of  all 
aircraft  of  the  same  model  produced  during  the  same  time  period 
(year) . 

•  Aircraft  age  can  be  managed  in  years  (age  of  the  airframe)  vice 
its  actual  flying  hours. 

•  Aircraft  are  paid  for  in  budget  year  t  and  delivered  in  budget 
year  t  +  L,  where  L  is  the  lag  time  for  production. 

•  All  expenditures  are  planned  for  in  constant  dollars. 

•  Monies  not  committed  in  budget  year  t  are  not  carried  forward 
to  subsequent  years . 

•  Annual  requirements  for  aircraft  include  float,  training  and 
operational  needs. 

•  Fixed  costs  associated  with  opening,  maintaining  and  closing  an 
aircraft  production  line  are  significant  and  must  be  included 
in  long-range  plans. 

•  Only  one  production  line  may  exist  for  a  particular  type  and 
model  of  aircraft  in  any  time  period.  Certain  models  are 
predecessors  to  others  on  an  individual  production  line  and 
can't  be  produced  concurrently,  while  production  lines 
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producing  different  aircraft  may  operate  simultaneously  in  a 
time  period. 

Production  lines  have  minimum  and  maximum  sustaining  rates  of 
production. 

The  user  of  the  model  specifies  different  requirement, 
resource,  and  policy  parameters  which  are  included  in  the 
constraints  of  the  model  and  which  can  influence  its  solution  (p. 
5.1).  The  requirement  parameters  are: 

•  The  number  of  time  periods  to  consider. 

•  The  minimum  and  maximum  number  of  aircraft  necessary  to  satisfy 
each  mission  in  each  time  period. 

The  policy  parameters  are: 

•  The  maximum  useful  life  (time  periods)  of  each  aircraft  in  a 
mission  fleet  in  the  model. 

•  The  minimum  fraction  of  each  mission  fleet  to  be  composed  of 
high-technology  aircraft  in  each  time  period. 

•  The  maximum  average  age  of  each  aircraft  in  a  mission  fleet  in 
each  time  period. 

The  resource  parameters  are : 

•  The  minimum  and  maximum  budget  available  to  spend  on  aircraft 
procurement,  aircraft  operations  and  maintenance,  and  on 
aircraft  retirement  in  each  time  period. 

•  The  existing  inventory  (including  year  of  manufacture  and 
number  of  aircraft)  and  their  technology  (high  or  low)  and  cost 
characteristics . 

•  The  technology  and  cost  characteristics  of  new  aircraft  designs 
and  design  improvements. 
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•  The  production  line  characteristics,  including  fixed  costs, 
production  capacities  and  limits,  types  of  aircraft  which  can 
be  produced  and  the  constraints  in  opening  and  closing  dates. 

D.  IMPORTANCE  OF  MODELING  STRATEGY  TO  ANALYSIS  OF  PHOENIX 

PHOENIX  is  a  real,  complex,  specific,  important,  contemporary, 
subjective  model.  Decisions  that  are  made  based  on  its  results  can 
influence  the  spending  of  billions  of  dollars.  Brown  et  al .  (1991) 
discuss  some  of  the  difficulties  encountered  during  the  modeling  and 
analysis  of  this  problem.  Many  are  problems  that  would  have  been 
alleviated  by  use  of  the  modeling  strategy  described  here.  In 
particular,  they  discuss  the  complications  faced  in  both  gathering 
and  validating  data  from  multiple  sources  and  in  uniting  this  data 
with  the  actual  model  formulation. 

The  selection  of  a  tangible  and  realistic  objective  function 
was  difficult  in  light  of  the  many  different  criteria  for  mission 
success  and  budget  limitations.  They  also  had  the  difficult  task 
of  developing  a  penalty  system  that  accurately  reflected  the 
relative  importance  of  each  of  the  measures  of  mission 
accomplishment  incoirporated  into  the  model  as  elastic  constraints. 
The  assignment  of  many  constraints  was  subjective  with  the  analysts 
using  "corporate  wisdom  . . . [to]  characterize  the  current  fleet 
status,  costs,  and  likely  consequences  of  future  procurement  and 
manufacturing  options"  (Brown  et  al,  1991).  Since  this  was  a 


29 


complex  problem  that  had  never  been  modeled  before  and  because  its 
impact  would  be  so  far-reaching,  there  was  great  concern  about 
ensuring  all  relevant  parameters  were  accounted  for  and  that  the 
model  was  a  true  depiction  of  the  problem.  The  situation  was 
further  compounded  because  the  model  developers  were  working  on 
separate  parts  of  the  model  on  opposite  sides  of  the  country. 

The  scope  of  the  PHOENIX  decision  is  broad,  influencing  not 
only  the  multi-year  planning  of  Army  helicopter  procurement  but 
interacting  with  the  modernization  of  other  Army  and  Department  of 
Defense  organizations.  The  helicopter  program  is  competing  for  a 
portion  of  a  fixed  budget.  PHOENIX  is  a  high  visibility  model  that 
must  be  justified  at  all  levels,  from  user  to  congressional,  and 
presented  to  a  variety  of  audiences  from  technicians  and  analysts 
to  generalists  in  Congress. 

Because  of  its  importance  and  the  dollars  involved,  it  is 
critical  that  the  analyst  explore  a  large  variety  of  alternate 
solutions  obtained  by  changing  the  resource,  policy  and/or 
requirement  parameters.  It  is  necessary  to  fully  analyze  each  of 
the  plans  since  one  alternative  could  provide  a  lower  objective 
function  value  than  another  plan,  but  not  be  considered  a  better 
plan  due  to  the  constraints  that  were  violated  and  the  corresponding 
money  assessed  in  penalties.  A  keen  understanding  of  the 
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implications  involved  in  each  of  the  alternate  plans  is  critical  to 
the  solution  of  the  problem. 

Sixteen  instances  of  the  model  were  run  as  part  of  the  initial 
study  and  two  were  presented  in  the  report.  Results  of  each 
scenario  had  to  be  put  into  tables,  then  converted  into  graphs. 
The  goal  of  the  strategy  presented  here  is  to  use  the  principles 
of  analysis  and  visualization  to  augment  the  validation  and  analysis 
process.  More  runs  could  be  analyzed  with  more  conditions 
considered.  This  would  yield  greater  confidence  in  the  decision 
recommended  and  help  to  prepare  the  results  for  presentation  to 
their  various  users. 
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VI.  IMPLEMENTATION  OP  THE  STRATEGY 


A .  TOOLS 

The  analysis  and  visualization  principles  presented  in  Chapter 
II  are  implemented  through  a  series  of  spreadsheets  and  the 
associated  graphics.  This  particular  application  is  built  using 
IMPROV  spreadsheets  and  Presentation  Builder  graphics  on  the  NEXT 
computer,  a  low  cost  ($5000)  workstation.  Other  visualization 
software  could  be  used  subject  to  user  preference  and  availability. 
The  specific  software  packages  are  not  important,  rather  the 
flexibility  and  enhanced  capabilities  that  they  render  allow  more 
time  to  be  devoted  to  analysis  rather  than  to  data  manipulation. 
This  is  especially  important  given  a  recurring  model  such  as  PHOENIX 
which  acts  as  a  decision  aid  in  a  continuously  updated  planning 
process . 

The  analyst  can  customize  a  series  of  worksheets  and  graphs 
that  is  generated  each  time  the  optimization  is  run.  It  is 
envisioned  that  each  optimization  model  would  dictate  a  set  of 
spreadsheets  and  graphs  that  are  uniquely  suited  to  the  analysis  of 
that  project.  These  tools  become  part  of  the  optimization  process 
and  are  available  to  be  used  and  amended  as  necessary  during  the 
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analysis.  This  streamlines  the  analysis  process  when  multiple  runs 
of  the  model  are  conducted.  The  same  information  for  each  run  is 
in  the  formatted  set  of  worksheets  and  graphics,  facilitating  the 
analysis  process  and  the  comparisons  between  successive  runs. 

The  use  of  spreadsheets  and  their  associated  graphics  as  tools 
in  the  optimization  process  provides  significant  advances  in  the 
flexibility  of  the  analysis.  The  data  is  easily  manipulated  via 
simple  formulas  opening  possibilities  for  recognizi.'.g  new 
information  or  new  aspects  of  information  that  can  be  derived  from 
the  raw  input  data  and  the  results  of  the  model.  The  data  can  be 
combined  in  many  different  ways  or  separated  into  its  individual 
pieces  as  the  analyst  sees  fit.  In  this  way,  data  is  manipulated 
to  reveal  information. 

Both  the  worksheets  and  the  graphic  packages  are  three 
dimensional.  Additionally,  any  changes  to  the  data  in  a  spreadsheet 
ripple  through  all  associated  spreadsheets  and  graphics  enabling  the 
analyst  to  see  how  the  change  effects  the  other  aspects  of  the 
model.  This  can  be  used  as  a  stepping  stone  in  model  validation  and 
exploration  in  a  limited  "What  if?"  scenario.  The  results  of  the 
changes  may  indicate  to  the  analyst  what  changes  to  make  to  input 
data  or  constraints. 


33 


whereas  the  IMPROV  spreadsheets  and  Presentation  Builder 
graphics  are  exceptionally  capable,  they  do  not  as  yet  incorporate 
all  the  technology  that  is  currently  available  in  similar  software 
packages.  Since  this  technology  will  be  available,  it  is  included 
in  the  application  presented  in  the  next  section.  In  particular, 
hypertext,  a  software  system  that  supports  special  links  within  a 
single  window  and  between  pairs  of  windows,  is  applied.  Hypertext 
allows  a  user  to  enter  a  software  package  and  program  changes  to  it 
that  enhance  the  capabilities  of  the  software  and  tailor  it  to  the 
user's  needs.  Here  the  term  hypertext  will  refer  to  the  capability 
for  the  user  to  zoom  in  and  view  only  selected  information  in  a 
window  cr  easily  move  from  viewing  one  window  to  viewing  another, 
usually  by  mouse  clicking  on  a  special  button.  Additional  aspects 
to  hypertext  systems  are  discussed  in  Conklin  (1987) . 

An  application  of  the  analysis  and  visualization  principles  to 
a  sample  run  of  PHOENIX  with  a  20-year  planning  horizon  and  a  2% 
real  budget  growth  rate  demonstrates  the  modeling  strategy.  The 
thought  processes  and  decisions  an  analyst  might  make  as  part  of  the 
interactive  optimization  process  conceptualized  in  Figure  1  are 
described.  The  caption  under  each  graph  includes  the  principles 
that  the  graph  most  remarkably  illustrates.  Some  of  the  principles. 
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such  as  a  customized  model  and  data  access,  apply  to  all  the  graphs 
and  are  not  specifically  noted  in  each  individual  graph. 

B.  VALIDATION  AND  VERIFICATION 

Of  initial  importance  to  the  analyst  is  the  assurance  of  the 
validity,  accuracy  and  consolidation  of  the  input  data.  The 
spreadsheet  environment  is  one  of  the  most  organized  and  effective 
means  to  consolidate  and  compare  data.  Data  fields  can  be  directly 
imported  into  a  spreadsheet.  Other  programs,  most  importantly  in 
this  case,  the  optimizer,  can  directly  read  from  these  spreadsheets. 
The  spreadsheets  are,  in  turn,  directly  linked  to  graphics 
presentations  where  changes  made  to  data  in  the  spreadsheet  are 
automatically  communicated  to  the  graphics. 

Using  the  appropriate  graphs,  the  analyst  can  look  for  trends 
and  discrepancies  from  trends  as  well  as  outlier  values  in  the  data. 
It  is  generally  obvious  from  the  nature  of  the  data  what  trends 
should  be  exhibited  and  any  deviation  from  these  trends  should  be 
examined.  For  example,  budgets  over  time  tend  to  increase  or  remain 
constant,  so  any  short  term  decrease  would  be  a  signal  to  the 
analyst  for  further  exploration.  It  could  be  indicative  of  an 
actual  trend  or  it  may  simply  be  the  result  of  incorrect  data.  The 
labels  in  the  spreadsheets  facilitate  the  verification  of  isolated 
values  and  sparse  elements  of  information.  This  process  is 
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ultimately  more  effective  and  less  time-consuming  than  manual 
scrutiny  of  raw  data  files.  The  input  data  for  the  PHOENIX  model 
consists  of  both  isolated  values  and  sparse  elements;  those  that 
will  display  a  trend  and  those  that  are  unrelated  pieces  of 
information  that  need  to  be  individually  scrutinized.  For  example 
the  budget  limits  are  easily  verified  for  consistency  in  Figure  2. 

It  is  important  to  realize  that  all  dollar  figures  in  PHOENIX 
have  been  adjusted  for  inflation  and  are  displayed  in  1988  dollars. 
Thus,  it  is  the  assumption  in  this  run  of  the  model  that  there  will 
be  an  increase  in  the  budget  maximum  limit  over  the  time  horizon  of 
the  model  where  the  lower  limit  will  remain  constant  after  1995. 

Associated  with  the  budget  are  several  growth  rates  that  can 
affect  the  problem  solution.  They  are  the  budget  growth  rate,  the 
inflation  rate  and  the  O&M  growth  rate  displayed  in  Figure  3.  In 
addition  to  the  actual  values,  it  is  interesting  to  see  these 
magnitudes  in  relation  to  each  other  and  to  verify  that  this  is 
consistent  with  the  trends  in  current  government  spending. 

Three  other  areas  which  should  display  trends  are  in  the 
mission  capability  areas  of  maximum  age,  high-technology  fraction 
and  force  requirements,  shown  in  Figure  4.  Since  the  goal  of 
PHOENIX  is  to  ensure  that  the  Army  helicopters  are  sufficiently 
capable  of  performing  their  mission  in  the  future,  it  is  logical  to 
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assume  that  the  trends  should  be  for  helicopters  that  are  better 
equipped  technologically  with  a  lower  maximum  age.  These 
accomplishments  would  allow  for  a  reduction  in  current  force  size 
to  some  constant  level.  See  Figure  4.  Any  deviations  from  these 
trends,  such  as  the  slight  dip  in  high-technology  fraction  goals  in 
1999,  are  apparent  to  the  analyst  who  will  then  attempt  to  discover 
their  cause.  Line  graphs  such  as  these  vastly  reduce  the  amount  of 
time  devoted  to  data  validation  and  become  extremely  useful  in 
analysis  and  presentation  of  results  of  the  model. 

When  graphically  displayed,  time  series  data  such  as  the 
scheduling  information  shown  in  Figure  5  becomes  much  more 
informative.  Recalling  that  PHOENIX  demands  that  only  one 
production  line  be  open  at  a  time  for  each  aircraft  mission  and  that 
certain  lines  are  predecessors  to  others,  this  graph  is  a  ready 
reference  for  determining  when  each  production  line  may  open  and 
close. 

Some  other  information  which  requires  verification  is  not  as 
meaningful  when  depicted  graphically  since  it  shows  no  trends  or 
consistencies.  In  PHOENIX,  this  data  includes  information  on 
aircraft  such  as  purchasing  cost,  last  high-technology  year  and  the 
other  information  shown  in  Figure  6.  The  spreadsheet  format  greatly 
enhances  the  ability  of  the  analyst  to  both  understand  the 
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information  and  verify  its  accuracy.  The  analyst  also  has  access 
to  other  information  in  spreadsheet  format  such  as  production  line 
capacities  and  initial  force  compositions.  Simple  spreadsheet 
formulas  can  easily  be  used  to  compare  and  consolidate  information. 

After  this  validation  and  consolidation  is  complete  and  any 
necessary  corrections  made,  the  optimization  model  is  ready  to  run. 
Spreadsheets  and  graphs  are  again  utilized  to  display  the  results 
as  part  of  the  interactive  process.  The  particular  views  that  are 
used  here  have  been  selected  as  part  of  the  PHOENIX  optimization 
process  but  can  easily  be  manipulated  to  highlight  any  information 
which  is  of  interest. 

I 

! 

C.  SINGLE-RXJN  ANALYSIS 

Once  the  optimization  is  run,  the  analyst  begins  the  analysis 
of  overall  results.  The  progression  through  the  analytical 
hierarchy  may  vary  by  analyst  and  situation  and  the  visualization 
tools  can  be  tailored  to  accommodate  this.  Of  initial  interest  is 
the  objective  function  value.  Recall  that  in  PHOENIX  the  objective 

i 

function  is  to  minimize  the  sum  of  the  O&M  costs  and  the  penalties. 
Look  at  both  the  total  objective  function  value  and  its  components 

I 

in  Figure  7.  Due  to  the  multi-year  planning  horizon  of  PHOENIX,  the 
analyst  may  gain  more  insight  from  examining  the  annual  components 
of  the  objective  function  value  in  Figure  8.  At  this  point,  the 
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analyst  could  choose  to  proceed  in  one  of  the  two  directions 
depicted  in  the  diagram  in  Figure  9. 

In  an  attempt  to  determine  whether  the  model  is  accurate  and 
provides  the  "best"  solution,  the  analyst  must  assess  whether 
penalty  values  are  realistic  as  assigned  and  if  penalties  taken 
within  the  model  are  appropriate  in  finding  an  optimal  solution. 
If  the  penalties  are  a  large  portion  of  the  objective  function 
value,  further  scrutiny  of  penalties  is  indicated.  If  the  penalties 
are  a  small  portion  of  the  total  or  if  their  relative  weight  cannot 
be  determined  at  this  point,  it  may  be  more  enlightening  to  examine 
the  total  budget  and  force  structure  resulting  from  the  optimization 
run. 

From  Figure  8,  the  analysts  sees  that  the  O&M  values,  which 
reflect  the  real-world  costs  of  Operations  and  Maintenance,  remain 
relatively  steady  with  a  gradual  increase  towards  the  out-years  of 
the  run.  The  penalty  values  are  high  at  the  beginning  of  the  model 
as  might  be  expected  because  the  model  decisions  have  little  impact 
on  fleet  condition  in  the  earliest  years.  The  penalties  go  to  zero 
in  the  mid-years  then  become  somewhat  significant  in  the  second  half 
of  the  planning  horizon.  If  the  penalties  were  large,  this  would 
invite  further  analysis  but  because  of  their  relatively  low  values 
the  analyst  may  choose  to  first  explore  the  budget  branch. 
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The  analyst  begins  with  a  look  at  the  total  budget  expenditures 
of  the  run  in  relation  to  the  minimum  and  maximum  budget  constraints 
in  Figure  10.  Recall  that  these  constraints  are  elastic  and  incur 
penalties  when  violated.  From  Figure  10  the  budget  exceeded  the 
maximum  in  the  first  two  years  then  stayed  within  budget  limits  for 
the  remainder  of  the  run,  dropping  quickly  in  the  out-years.  A 
closer  look  at  expenditures  is  therefore  warranted. 

Although  penalties  are  important  to  the  solution  of  the  model, 
the  actual  annual  expenditures  are  composed  of  only  the  procurement 
and  O&M  costs.  Figure  11  reveals  these  actual  expenditure  figures 
along  with  minimum  and  maximum  budget  limits.  The  last  few  years 
of  a  model,  in  this  case  2007  and  2008,  may  not  be  valid.  This  is 
a  well-known  aspect  of  such  multiple-time  period,  finite  horizon 
models  with  fixed  enulng  and  beginning  conditions.  It  is  made  much 
more  obvious  by  the  visualization  techniques  used  in  this  analysis. 
This  anomaly  is  caused  by  the  optimizer's  approach  to  minimizing 
costs  and  incurring  penalties  towards  the  model's  completion  rather 
than  investing  in  expensive  procurement  actions  whose  long-range 
pay-off  would  not  be  realized  within  the  scope  of  the  model.  This 
tendency  must  be  considered  in  viewing  and  implementing  the  results 
of  the  model.  Closer  examination  of  Figure  11  indicates  a  leveling 
off  of  procurement  from  the  year  2000  and  beyond  even  though  annual 
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expenditures  are  further  and  further  below  the  increasing  maximum 
budget.  The  analyst  would  explore  the  various  explanations  for  this 
trend  by  asking  questions  such  as  "Is  the  model  capable  of  meeting 
all  goals  with  less  than  total  expenditure?"  and  "Is  the  full 
budgeted  amount  not  required?". 

In  order  to  answer  these  questions,  the  analyst  must  first 
determine  the  answers  to  more  basic  questions  such  as  "What  did  we 
get?"  or  "What  did  we  not  get?".  The  first  question  could  be 
answered  by  looking  at  force  compositions,  supplemented  by  age  and 
hi-tech  compositions.  In  this  instance,  however,  the  question  of 
what  goals  were  not  met  by  the  solution  is  more  instructive.  It  is 
best  answered  by  examination  of  the  penalty  data. 

Switching  focus  to  the  penalty  branch  of  the  analytical 
hierarchy  in  Figure  9,  the  analyst  would  see  which  goals  are  not 
being  met.  Figure  12  shows  the  amount  and  type  of  penalty  assessed 
in  each  year  of  the  model.  It  is  understandable,  but  not  very 
controllable  within  the  model,  that  significant  penalties  are 
assessed  in  the  first  two  model  years.  Their  magnitude  somewhat 
conceals  the  breakdown  of  penalties  in  the  later  years  of  the  model 
which  are  of  greater  interest  in  the  optimization  process.  In 
order  to  focus  on  the  later  penalties  and  reveal  the  information, 
the  analyst  can  use  the  hypertext  technique  to  zoom  in  on  the  years 
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of  interest.  By  hiding  1989  and  1990  a  new  graph.  Figure  13,  is 
created. 

In  a  similar  way,  at  any  time  the  analyst  may  see  fit  to  focus 
on  segments  of  the  data  that  may  be  of  significant  concern  or 
interest,  for  example  a  segment  of  years  where  goals  are  not  met, 
a  particular  role  of  helicopters,  etc.  The  graphical  hierarchy  and 
use  of  a  hypertext  technique  allow  the  analyst  to  move  about  within 
the  structure  and  zoom  in  on  a  selected  period  of  time  or  other 
factor . 

At  this  stage  if  the  penalties  appear  to  be  extreme,  the 
analyst  may  again  question  the  fidelity  of  the  model.  A  return  to 
the  data  and  problem  definition  may  be  warranted  to  assess  the  need 
for  making  changes  in  the  penalty  structure.  In  this  case, 
however,  the  penalties  do  not  appear  extreme  and  the  analysis 
process  continues. 

The  analyst  sees  in  Figure  13  a  surge  of  high-technology 
penalties  in  the  middle  years  of  the  model,  tapering  off  to  almost 
nothing.  Mission  requirement  penalties  increase  significantly 
beginning  in  1997  then  decrease  somewhat,  and  there  is  a  steady  rise 
in  age  penalties  in  the  later  years. 

Since  the  requirements  for  high-technology,  average  age  and 
mission  requirements  are  different  for  each  of  the  four  aircraft 
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roles  --  attack,  scout,  cargo  and  utility  --  it  is  important  to  view 
the  penalties  as  they  effect  each  role.  This  reveals  the  specific 
areas  that  need  further  investigation.  Figure  13  shows  that  any 
real  trends  in  penalties  will  be  found  in  years  1997  to  2008.  The 
analyst  again  focuses  on  these  years  in  the  three  appropriate 
penalty  areas.  Figure  14. 

It  appears  that  Scout  aircraft  are  primarily  responsible  for 
both  the  mission  requirements  penalties.  Graph  14a,  and  the  high- 
technology  penalties.  Graph  14b,  especially  from  1998  to  2004.  In 
later  years.  Cargo  aircraft  are  responsible  for  most  of  the 
remaining  penalties  in  the  categories  of  mission  requirements  and 
age. 

It  is  now  necessary  to  look  in  even  greater  detail  at  each 
penalty  category  and  to  assimilate  information  from  different  parts 
of  the  model.  The  windows  environment  facilitates  this  part  of 
the  process  by  allowing  multiple  visualization  tools  to  be  displayed 
on  the  screen  simultaneously.  For  example,  in  considering  the 
mission  requirement  penalties,  it  is  instructive  to  simultaneously 
view  the  force  composition  graphs  and  the  penalty  information  as  in 
Figure  15. 

The  analyst  is  particularly  interested  at  this  juncture  in  the 
Scout  and  Cargo  helicopters.  In  the  years  where  penalties  were 
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assessed,  the  Scout  force  is  composed  mainly  of  OH-58D's,  with  the 
OH-58A's  and  OH-58C's  having  been  SLEPed  into  58D's.  Production  of 
LHX-SCT's  begins  in  the  model  in  the  year  2002,  at  which  point  the 
build-up  returns  the  Scout  force  to  its  minimum  requirements  and 
penalties  are  no  longer  being  assessed  in  this  category.  This  is 
all  clearly  visible  to  the  analyst  from  Figure  15. 

It  is  important  for  the  analyst  to  examine  the  reasons  that  the 
optimization  chose  this  particular  course  of  action  and  to  consider 
such  questions  as,  “Why  was  the  LHX-SCT  not  procured  sooner  to 
avoid  these  penalties?"  and  "Is  it  due  to  production  limits,  budget 
limitations,  or  was  it  driven  by  O&M  costs  in  the  objective 
function?".  First,  however  the  analysis  will  continue  with  an 
examination  of  the  other  force  composition  penalties  at  this  same 
level , 

The  force  composition  of  Cargo  aircraft  from  1997  on  is 
primarily  CH-47D's.  There  is  no  indication  of  production,  and  the 
model  continues  to  accrue  penalties  in  force  composition  for  the 
duration  of  the  model.  Again,  "Why  are  these  penalties  incurred  and 
what,  if  any,  alternatives  are  available?".  Beginning  with 
production  limitations  as  a  possible  explanation,  the  analyst 
examines  the  graph  of  production  line  schedules  and  observes  from 
Figure  16  that  the  production  capability  for  Cargo  helicopters  must 
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close  by  1994.  It  would  not  be  cost  effective  to  begin  production 
earlier  since  the  force  structure  of  Cargo  helicopters  is  adequate 
through  2003  and  there  is  no  production  capability  after  1994.  This 
answers  one  chain  of  the  single-run  analysis  and  could  be  noted 
for  additional  exploration. 

Returning  to  Figure  14,  the  analyst  proceeds  with  an 
examination  of  the  high-technology  penalties  which  are  due  almost 
entirely  to  Scout  aircraft,  beginning  in  1999  and  tapering  off 
dramatically  at  the  end  of  the  model.  By  examining  the  force 
structure  broken  down  by  role  in  Figure  15,  the  analyst  can  see  that 
all  the  existing  Scout  aircraft  switch  from  high  to  low  technology 
in  1999.  The  Force  Composition  graph  indicates  that  Scout  force  is 
almost  entirely  made  up  of  OH-58D's.  The  use  of  hypertext  would 
allow  immediate  access  to  the  information  behind  the  graph.  The 
analyst  would  be  able  to  view  and  assimilate  all  this  information 
quickly  and  easily  by  retrieving  these  graphs  onto  the  screen  and 
creating  an  environment  like  Figure  17. 

Once  LHX  production  begins  in  2002,  the  model's  minimum 
fraction  high-technology  limit  is  almost  attained  for  Scout 
helicopters  by  the  final  year  of  the  model.  The  analyst  may  wonder 
what  measures  would  have  avoided  the  penalties.  Another  look  at  the 
production  schedule  and  associated  spreadsheet  information  shows 
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that  LHX  production  could  begin  as  early  as  1995.  Figure  18  shows 
Attack  LHX  helicopters  were  procured  as  early  as  1997  and  the 
production  line  was  capacitated  for  the  first  two  production  years. 
Note  that  the  force  composition  graphs  do  not  show  LHX  Attack 
helicopters  in  the  fleet  until  1999.  This  is  due  to  the  lag  years 
between  purchasing  and  delivery.  Clearly  it  is  more  cost-effective 
to  procure  Attack  aircraft  and  pay  penalties  for  shortfalls  in  the 
Scout  fleet  than  to  procure  Scouts. 

Investigation  of  the  age  penalties  in  Figure  14  shows  the 
analyst  that  all  penalties  are  due  to  Cargo  aircraft,  beginning  in 
year  2004  and  escalating  to  the  end  of  the  model.  A  simultaneous 
look  at  the  production  schedule,  the  maximum  age  and  force 
composition  graphs.  Figure  19,  shows  that  there  is  no  production 
capability  for  Cargo  helicopters,  H-47's,  in  the  later  half  of  the 

The  analysis  process  is  controlled  by  the  user  and  the  specific 
approach  will  depend  on  the  results  of  the  model  and  the  questions 
to  be  answered.  All  of  the  input  data  and  the  results  are  available 
in  the  spreadsheets  and  graphs.  The  way  the  analyst  chooses  to 
navigate  through  this  information  is  subjective.  Any  train  of 
thought  or  process  of  inquiry  can  be  followed.  Once  the  analyst  has 
fully  explored  a  single  run,  it  is  then  possible  to  begin 
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sensitivity  analysis  by  comparing  the  solution  with  one  or  more 
subsequent  runs  of  the  model . 

D.  MULTIPLE  RUN  ANALYSIS 

A  single  optimal  solution  is  seldom  acceptable  for  a  complex 
optimization  problem.  No  solution  can  account  for  all  possible 
contingencies.  As  many  contingencies  as  possible  need  to  be 
considered  within  the  realm  of  analysis.  The  developers  of  PHOENIX 
recognized  this  and  included  as  one  of  their  findings,  "Mixed 
integer  linear  programming  provides  no  useful  dual  information  for 
conducting  postoptimality  analysis.  Sensitivity  analysis  of  model 
output  must  be  accomplished  using  multiple  runs  with  varying  input 
data."  (Force  Systems  Directorate,  1988). 

In  the  final  report  on  the  PHOENIX  model  (Force  Systems 
Directorate,  1988),  two  solutions  were  provided,  each  solved  by  the 
same  model  with  the  same  parameters  except  for  the  budget  growth 
rate.  Fourteen  other  runs  were  made,  but  the  results  were  not 
provided  (p.  3-10) .  Since  the  goal  of  the  study  was  “to  formulate 
and  implement  a  prototype  decision  aid  for  force  planners  for 
evaluating  the  effect  of  aviation  modernization  policy  over  an 
extended  planning  horizon"  (p.  1-1),  the  report  did  not  emphasize 
the  solutions  of  PHOENIX  but  rather  its  role  as  a  decision  aid.  The 
application  of  analysis  and  visualization  principles  presented  here 
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would  have  greatly  enhanced  the  ability  of  the  modelers  to  analyze 
and  report  on  these  solutions.  Additionally,  current  users  of  this 
model  would  better  conduct  sensitivity  analysis  by  making  the 
multiple  runs  and  evaluating  and  comparing  their  results  using  this 
modeling  strategy. 

This  section  will  discuss  two  ways  to  perform  multiple  run 
analysis  and  apply  each  to  the  PHOENIX  model.  The  first  scenario, 
which  is  probably  the  more  common,  solves  a  baseline  model  with  the 
best-known  parameters.  The  analyst  may  want  to  explore  how 
relaxations  or  changes  to  these  parameters  effect  the  solution. 
This  is  typical  in  many  optimization  situations,  especially  in  ones 
like  PHOENIX  that  are  time  sequence  problems  with  multiple  measures 
of  effectiveness  in  the  objective  function  and  constraints. 

The  second  approach  also  involves  the  changing  of  parameters, 
however  it  does  not  presume  that  one  set  best  represents  the 
solution.  Rather  it  may  compare  a  variety  of  solutions  obtained  by 
systematically  changing  one  or  several  parameters,  such  as  inflation 
rate  or  budget  growth  rate.  Although  this  approach  was  not 
originally  reported  in  the  PHOENIX  study,  it  is  practical  and 
useful,  especially  when  considering  a  long  term  planning  process  in 
an  era  of  ever-changing  government  spending  priorities.  Many  of  the 
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same  graphical  techniques  can  be  applied  to  both  cases  although  they 
may  be  more  instructive  for  one  purpose  over  another. 

1.  General  Comparisons 

One  of  the  most  important  aspects  of  comparisons  of 
solutions  is  the  comparison  and  documentation  of  the  parameters  that 
change.  Since  the  input  data  for  each  run  of  the  model  is  displayed 
in  graphs  or  tables,  it  is  relatively  simple  to  produce  side-by- 
side  comparisons  of  these  values.  Highlighting  the  information  that 
is  different  is  also  accomplished  very  easily  in  these  environments. 
Figure  21  demonstrates,  for  a  two  case  scenario,  how  effective  the 
side-by-side  comparison  can  be  in  both  graphical  and  tabular  form. 
Figure  21a  shows  the  different  objective  function  values  and  their 
breakdown  into  O&M  costs  and  penalties.  The  relative  contributions 
of  each  of  these  components  as  well  as  the  actual  objective  function 
values  is  quite  apparent.  Figure  21b,  on  the  other  hand,  shows  the 
change  in  the  purchasing  cost  of  LHX  aircraft,  a  change  that  can 
drastically  alter  the  outcome  of  the  solution,  displayed  in 
spreadsheet  form. 

2.  Comparison  to  Base  Solution 

The  objective  function  value  is  typically  one  of  the  most 
important  criteria  for  comparison  of  solutions.  However,  this  is 
not  always  the  best  criteria.  In  a  model  like  PHOENIX  which  has 
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elastic  constraints  and  associated  penalties  that  are  incorporated 
as  part  of  the  objective  function  value,  careful  consideration  must 
also  be  given  to  the  rest  of  the  solution.  It  is  important  to  note 
which  goals  have  been  met  and  which  have  not  been  met  in  each 
solution  over  the  time  period  of  the  model.  It  is  also  important 
to  look  at  the  differences  between  each  of  the  models  for  each  of 
the  constraints.  For  example.  Figure  22  shows  a  comparison  of  the 
total  annual  expenditures  of  two  runs  of  the  model.  The  line  graph. 
Figure  22a,  indicates  the  actual  expenditures  in  the  two  cases  but 
Figure  22b  is  more  enlightening  as  it  is  a  graph  of  the  differences 
in  the  two  solutions.  It  clearly  shows  that  the  expenditures  of  the 
second  run  are  consistently  lower  than  those  of  the  base  case. 
These  same  types  of  graphs  can  be  constructed  for  force 
compositions,  procurements  and  the  other  constraint  conditions. 
They  can  also  be  used  for  determining  the  differences  in  the 
penalties  that  are  assessed  in  each  situation. 

Another  type  of  graph  that  can  be  used  to  compare  results 
of  two  runs  is  the  scatterplot  depicted  in  Figure  23,  which  displays 
the  same  information  as  Figure  22a.  Clearly,  any  deviations  from 
the  identity,  or  x=y,  line  indicate  differences  in  the  two 
solutions.  A  majority  of  the  points  to  the  right  of  or  below  the 
x=y  line,  as  in  Figure  23,  indicates  larger  values  for  the  solution 
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associated  with  the  horizontal  axis,  in  this  case  for  the  base 
solution.  If  points  were  scattered  on  either  side  of  the  line, 
neither  solution  would  be  generally  better  or  worse. 

When  comparing  several  different  solutions  against  a  base 
case  by  display  in  a  small  multiples  format,  these  graphs  become 
quite  dramatic  and  informative.  The  eye  is  able  to  quickly  make 
comparisons  and 

inadequate  solutions  may  sometimes  be  discarded. 

3.  Comparison  of  Several  Solutions 

The  use  of  multiples  described  above  also  works  very  well 
in  the  case  of  the  comparison  of  several  solutions.  This  is 
demonstrated  in  Figures  24  and  25.  Figure  24  is  a  multiple  display 
of  the  objective  function  values,  broken  into  O&M  costs  and 
penalties.  The  eye  can  focus  on  the  differences  in  the  data  and 
the  mind  is  able  to  quickly  assimilate  this  information.  It  is  much 
more  effective  than  having  to  turn  pages  or  even  look  at  two  pages 
simultaneously.  Figure  25  is  an  even  more  striking  display  of  the 
objective  function  values.  While  it  does  not  give  quite  as  much 
information  as  Figure  24,  the  boldness  of  the  display  and  the  close 
proximity  of  the  graphs  to  each  other  makes  assessments  of  their 
relative  values  very  apparent . 


51 


other  data  analysis  techniques  can  also  be  used  to  compare 
several  solutions.  The  use  of  color  serves  very  well  in 
distinguishing  between  several  solutions.  A  single  color  associated 
with  each  particular  solution  lends  consistency  to  the  design  and 
can  serve  to  highlight  the  best  solution  or  at  least  narrow  down  the 
field  to  a  few  solutions  that  can  be  further  analyzed.  Even  when 
color  is  not  available,  the  use  of  different  shades  of  gray  or 
different  textures  serve  the  same  purpose,  as  shown  in  the  PHOENIX 
application. 

When  dealing  with  many  possible  solutions,  it  is 
instructive  to  use  some  statistical  tools.  For  example,  when 
discussing  the  force  composition  of  attack  aircraft  in  each  year  of 
the  model,  it  may  be  useful  to  consider  the  minimum,  maximum  and 
average  number  of  aircraft  for  each  year  from  all  the  runs  of  the 
model.  This  could  be  done  quite  efficiently  with  boxplots.  Figure 
26.  These  plots  can  indicate  the  distribution  of  the  data  and  show 
any  skewing.  These  boxplots  could  be  used  to  analyze  constraints. 
If  a  display  of  the  results  of  multiple  runs  of  the  PHOENIX  model, 
for  example  the  number  of  attack  aircraft  of  a  certain  type  procured 
in  each  year,  showed  little  variation,  this  could  indicate  to  the 
analyst  that  this  constraint  might  be  tightened  or  restricted  in 
some  way  without  effecting  the  rest  of  the  model. 
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E.  PRINCIPLES  APPLIED  TO  PHOENIX 

The  graphs  and  spreadsheets  of  the  previous  sections  illustrate 
the  application  of  particular  analysis  and  visualization  principles 
to  the  PHOENIX  model.  There  are,  however,  some  general  principles 
that  apply  to  all  the  graphs. 

One  of  the  goals  of  visualization  is  to  reveal  the  information 
contained  in  the  data.  The  eye  should  focus  on  the  information  of 
interest.  The  graphs  are  created  to  bring  any  outlying  values, 
major  discrepancies  or  trends  into  the  forefront. 

Revealing  constraint  violations  is  also  important.  This  is 
done  by  combining  bar  and  line  charts.  The  actual  values  that  the 
solution  produces,  such  as  the  force  composition  of  attack  aircraft, 
are  of  primary  interest,  but  the  analyst  also  needs  to  see  how  this 
number  compares  with  its  goal.  One  piece  of  information  complements 
the  other  but  does  not  overpower  it  as  it  can  easily  be  ignored  if 
not  of  immediate  interest. 

Some  information  is  duplicated  in  a  single  graph.  This  is  the 
case  in  Figure  11,  the  annual  expenditures.  Both  the  height  of  the 
bar  and  the  text  at  the  top  of  the  bar  contain  the  same  data.  This 
was  done  because  of  both  the  magnitude  of  the  information  and  its 
importance  to  the  overall  solution. 
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Another  valuable  aspect  of  these  graphs,  and  one  of  the 


visualization  principles,  is  consistency.  Graphs  of  comparable 
values  are  consistent  in  size,  coloring  and  labeling.  The  order  of 
aircraft,  both  individually  and  by  role,  also  stays  the  same.  This 
consistency  is  important  to  the  analyst  because  it  reduces  the 
amount  of  time  spent  interpreting  the  data  in  favor  of  time  spent 
analyzing  it.  It  also  facilitates  combining  similar  information, 
such  as  the  number  of  Scout  and  Attack  aircraft  procured  in  a  year, 
which  lends  a  different  view  to  the  analysis  process. 

Finally,  the  graphs  are  designed  to  highlight  and  explore  the 
relationships  among  the  data  values.  This  is  accomplished  through 
the  windowing  environment  which  allows  the  overlaying  of  one  graph 
upon  or  next  to  another.  The  relationships  can  also  be  explored  by 
the  display  of  data  values  themselves.  These  values  are  always 
available  in  easily  accessible  spreadsheets.  The  hypertext 
capability  will  ensure  these  are  available  at  the  click  of  a  button. 

There  are  also  other  features  not  currently  included  in  this 
analysis  due  to  limited  technology  but  which  could  be  developed  to 
be  of  great  assistance.  One  technique  is  to  capture  the  thought 
processes  of  the  analyst  in  a  scripted  scenario  that  could  later  be 
reviewed.  This  could  be  of  significance  when  a  model  is  only  used 
periodically.  If  the  analyst  decides  to  update  or  review  the  plan. 
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it  would  be  invaluable  to  be  able  to  see  the  process  by  which 
analysis  was  initially  conducted  and  how  the  decisions  were  made 
when  the  model  was  originally  run.  In  many  circumstances  a  new 
analyst  might  be  updating  the  plan.  That  analyst  could  follow  the 
scripted  version  of  the  initial  run  and  gain  much  insight  and 
understanding  prior  to  conducting  follow-up  runs  and  making  changes. 

This  scripted  version  could  be  designed  to  lead  the  analyst 
through  the  input  data  and  results  in  the  same  order  as  was 
originally  done.  Voice  narration  could  even  be  included  to  further 
document  the  decisions.  The  graphs  could  appear  in  a  flip-chart 
format ;  they  could  fade  in  and  out  or  they  could  be  programmed  to 
open  and  close  as  appropriate.  This  script  could  also  be  used  by 
the  analyst  to  present  decisions  or  options  to  planners. 
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VII.  THE  FtTrURB  OP  PHOENIX 


When  it  was  first  implemented  in  early  1988  PHOENIX  provided 
the  Army  with  a  detailed  plan  for  the  Army  Aviation  Modernization 
Program  (AAMP.  This  plan  was  revealed  in  a  press  release  by  the 
Secretary  of  the  Army  dated  September  30,  1988  (Brown  et  al,  1991) 
which  stated: 

The  funding  provides  for  an  efficient,  cost-effective 
production  rate  of .. .aircraft  in  quantities  required  by  the 
Army's  force  structure  in  meeting  the  requirements  of  the 
unified  and  specified  commanders-in-chief,  and  to  achieve  an 
optimum  program  within  the  funding  constraints. 

In  essence,  PHOENIX  has  guaranteed  a  modern  fleet  of  helicopters 

within  budget  that  will  carry  the  Army  through  the  next  25  years  and 

beyond.  Since  that  time,  PHOENIX  has  also  been  successfully 

modified  and  adopted  for  use  in  the  force  planning  of  tactical 

wheeled  vehicles.  Additionally,  PHOENIX  has  been  expanded  to 

include  production  of  component  parts  and  subassemblies  of  major 

systems  as  well  as  transfers  from  one  force  package  to  another.  A 

force  package  is  a  collection  of  units  grouped  together  based  on 

deployment  schedules  such  as  active  versus  reserve  forces. 

(Coblentz,  1991) 
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In  addition  to  PHOENIX,  CAA  uses  another  force  planning  tool, 
the  Force  Modernization  Analyzer  (FOMOA) .  It  is  a  scaled-dovm  non¬ 
integer  linear  program  version  of  PHOENIX.  It  is  implemented  in 
spreadsheet  form  on  a  Macintosh  personal  computer  using  the  Super 
MacVino  optimization  package.  It  is  designed  for  use  as  a  quick 
reaction  analysis  tool  and  has  a  turnaround  time  of  approximately 
twenty  minutes. 

These  two  force  planning  tools  are  used  to  complement  one 
another.  PHOENIX  is  the  far  more  capable  model  but  it  requires 
well-trained  analysts.  FOMOA  does  not  require  any  mathematical 
programming  skills  but  does  require  user  input  of  such  decisions  as 
which  production  lines  will  open.  The  constraints  of  FOMOA  are  a 
subset  of  the  constraints  in  PHOENIX  and  cannot  be  violated.  The 
combination  of  time,  data  resources  and  level  of  detail  required 
dictate  which  model  is  used  in  each  circumstance.  (Coblentz,  1991) 

PHOENIX  requires  significantly  more  input  data  and  its  results 
require  more  extensive  analysis  than  FOMOA.  This  is  one  of  the 
inherent  differences  between  the  two  models  and  cannot  be  changed 
without  changing  their  purposes.  One  of  the  primary  advantages  of 
FOMOA  is  its  quick  turnaround  time.  PHOENIX  originally  required 
approximately  10-12  minutes  of  computer  processing  (CPU)  time  for 
a  single  run,  but  now  runs  in  one  to  two  minutes.  However,  CAA 
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does  not  currently  have  the  computer  capability  to  support  a  run  of 


PHOENIX  so  it  must  be  run  on  a  computer  at  the  Pentagon.  This  adds 
considerable  time  to  the  run  of  the  model,  not  only  physical  commute 
time,  but  also  time  spent  waiting  for  processing  by  the  Pentagon 
computer . 

The  scheduled  addition  of  a  workstation  to  CAA  headquarters 
will  give  them  the  computer  capability  to  solve  the  PHOENIX  model. 
While  it  takes  more  CPU  time  on  a  workstation  than  on  a  mainframe, 
the  total  time  required  to  conduct  a  run  of  PHOENIX  should  decrease 
as  it  would  no  longer  have  to  compete  with  other  Pentagon  projects 
for  processing  and  the  commute  will  be  eliminated.  The  addition  of 
an  in-house  workstation  will  reduce  the  time  to  solve  PHOENIX  from 
days  to  minutes. 

Application  of  the  analysis  and  visualization  principles  to  the 
PHOENIX  model  as  demonstrated  in  the  previous  chapter  would 
similarly  reduce  the  time  required  to  conduct  a  full  run  of  the 
PHOENIX  model,  from  data  validation  to  multiple  run  analysis.  After 
the  results  of  a  solution  run  are  entered  into  the  spreadsheet,  all 
the  worksheets  and  graphs  described  in  the  previous  chapter  are 
constructed  in  a  matter  of  seconds. 

This  combination  of  a  workstation  environment  and  the  modeling 
strategy  would  bring  some  of  the  convenience  of  FOMOA  to  the 
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sophistication  of  PHOENIX.  Problems  that  require  the  details  of  the 
PHOENIX  model  would  be  solved  faster  and  more  efficiently.  It 
would  allow  more  thorough  analysis  of  multiple  runs  in  a  shorter 
time  and  would  allow  more  runs  to  be  made  in  order  to  explore  a 
greater  number  of  alternate  optimal  solutions. 
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VIIX .  CONCLUSIONS 


The  widespread  availability  and  technological  advances  in 
computer  capability  and  the  power  of  information  in  today's  society 
support  the  steady  growth  of  mathematical  programming  and 
optimization  in  business,  industry,  government,  and  academia. 
Today's  real,  complex  and  important  problems  can  be  solved  through 
the  use  of  large-scale  optimization  with  a  fidelity  and  accuracy 
that  was  impossible  a  few  years  ago.  The  management  and  economic 
impact  of  using  operations  research  is  now  widely  recognized  and  the 
tools  and  techniques  are  being  implemented  extensively.  As  bigger 
problems  are  modeled,  the  voluminous  input  and  output  increase  the 
need  for  effective  analysis  while  complicating  its  execution, 
frequently  making  the  time  and  abilities  of  the  analyst  the  limiting 
factor  in  the  process.  The  ability  to  make  changes,  to  update  and 
explore  the  possibilities  of  the  model,  and  to  communicate  the 
results  to  the  people  who  have  the  problem  is  time  consuming  and 
difficult,  but  crucial  to  effective  analysis.  Through 
implementation  of  the  modeling  strategy  developed  in  this  thesis, 
these  obstacles  can  be  overcome  and  thorough,  effective  validation 
and  analysis  of  large-scale  optimizations  can  become  the  norm. 
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The  principles  of  analysis  and  visualization  outlined  as  the 
basis  of  this  modeling  strategy  are  tailored  to  a  specific  problem 
to  produce  the  hierarchical  structure  of  customized  tools,  such  as 
graphs  and  spreadsheets.  The  analyst  controls  the  process  and 
focuses  on  areas  of  interest.  The  application  of  appropriate 
displays,  simple  graphical  techniques,  representation  driven  graphs, 
and  consistency  between  representations  allows  the  analyst  to  more 
quickly  perceive  the  extensive  information  and  to  readily  identify 
trends,  outliers,  constraint  violations,  and  penalties  accrued  on 
elastic  constraints.  The  analyst  can  then  relax  constraints,  change 
penalties,  and  explore  a  variety  of  feasible  and  alternate  optimal 
solutions.  The  access  to  source  data  and  relationships  among  values 
and  the  backward  compatibility  of  this  modeling  strategy  facilitate 
this  analysis. 

Having  developed  the  structure  of  validation  and  analysis  tools 
for  the  particular  problem,  data  from  subsequent  runs  of  the 
optimization  can  be  ported  into  spreadsheets  and  the  same  analysis 
tools  created  for  this  data  in  a  matter  of  seconds.  Multiple 
representations  can  add  new  views  or  information  to  extend  the 
model.  Sensitivity  analysis  and  comparisons  of  multiple  runs  are 
significantly  enhanced  by  this  strategy  and  by  the  use  of  side- 
by-side  comparisons,  base  case  comparisons,  and  multiples. 
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This  modeling  strategy  is  effectively  applied  to  large-scale 
optimizations  in  general  but  is  especially  expedient  for  multi¬ 
year  and  recurring  models  and  the  complex  and  subjective  problems 
more  frequently  modeled  today.  The  benefits  of  creating  the  tools 
and  implementing  the  strategy  are  quickly  realized.  The  developed 
model  is  easily  learned  by  second-generation  and  subsequent  users 
and  is  easily  reviewed  between  uses. 

As  large-scale  optimization  models  play  an  increasing  role  in 
industry  and  government,  analysts  will  seek  better  ways  to  represent 
and  solve  complex  problems.  In  cases  where  an  analyst  would 
otherwise  not  be  able  to  thoroughly  assess  voluminous  data  and  a 
large  array  of  alternate  solutions,  implementation  of  this  strategy 
will  support  thorough  analysis  and  selection  of  the  best  possible 
decision. 
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APPENDIX 


Figure  1.  Interactive  Optimization  Process.  The  traditional 
optimization  process  is  indicated  by  blocks  and  forward  arrows.  The 
modeling  strategy  for  the  interactive  optimization  process 
emphasizes  the  analysis,  verification/consolidation, 
validation/fidelity  and  multiple  run  arcs. 
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Figure  2.  Budget  Constraints.  Verification  of  input  values  is  more 
readily  conducted  through  the  use  of  presentation  graphics.  As 
expected,  the  maximum  budget  input  in  this  case  is  steadily 
increasing  and  the  minimum  budget  input  levels  off  after 
stabilization  of  the  model.  This  type  of  graph  supports  the 
visualization  principles  of  simplicity  and  appropriate  displays. 


Figure  3.  Parameter  Growth  Rates.  Input  parameters  are  easily 
compared  using  a  simple  bar  graph.  The  visualization  principle  of 
simplicity  is  highlighted  here. 
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Figure  4.  Policy  Parameter  Input  by  Helicopter  Role.  Minimums  and 
maximums  should  appear  as  increasing  or  decreasing  functions.  The 
drop  in  Figure  4(b)  in  1999  alerts  the  analyst  to  a  possible  input 
error  for  Attack  helicopters.  The  visualization  principle  of 
consistency  is  portrayed  by  the  use  of  identical  line  types  for  type 
of  helicopter  in  each  graph,  facilitating  interpretation. 
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Figure  5.  Schedule  of  Production  Lines.  This  graph  is  useful  in 
the  validation  of  production  line  input  data.  it  also  will  be 
useful  in  the  analysis  of  the  model  to  find  causes  of  penalties  and 
force  composition  shortfalls.  It  demonstrates  the  visualization 
principle  of  data  access. 
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Figure  6.  Segment  of  Spreadsheet.  Verification  of  sparse  data  and 
isolated  values  is  easily  conducted  in  the  spreadsheet  environment. 
This  supports  the  visualization  principles  of  simplicity  and 
appropriate  displays. 
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Figure  7.  Total  Objective  Function  Value  and  Components.  This 
graph  shows  the  beginning  of  the  analysis  principle  of  hierarchical 
structure.  The  objective  function  value  is  comprised  of  the  O&M 
costs  and  the  penalties. 
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Figure  9.  Hierarchical  Structure  of  PHOENIX.  A  hierarchical 
structure  is  common  to  the  modeling  strategy  for  large-scale 
optimizations.  The  specific  elements  of  the  structure  would  vary 
by  specific  application. 


Figure  10.  Budget  Constraint  Violations.  The  simplicity  of  the 
line  graph  makes  it  easy  to  see  where  the  optimal  annual  budget 
violated  budget  minimums  or  maximums.  This  graph  also  pursues  the 
budget  branch  of  the  hierarchical  structure  and  represents  the 
visualization  principle  of  representation  driven  graphs. 
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Figure  11.  Annual  Expenditures  by  Component.  This  graph  continues 
in  the  hierarchical  structure  and  portrays  the  visualization 
principles  of  representation  driven  graphs,  appropriate  displays, 
and  simplicity. 
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Figure  12.  Total  Annual  Penalties.  Analysis  within 
hierarchical  structure  can  proceed  on  a  different  branch, 
demonstrated  by  switching  from  budget  data  to  penalty  data. 
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Figure  13.  Penalties  components  for  selected  years.  The  use  of  the 

zooming  principle  of  visualization  allows  the  analyst  to  view  in 
greater  detail  the  penalty  values  for  selected  years  of  interest. 
This  graph  also  highlights  the  principles  of  hierarchical  structure, 
consistency  and  appropriate  displays. 
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Figure  14.  Type  of  Penalty  Incurred  by  Aircraft  Role.  The 

hierarchical  structure  aids  in  the  identification  of  specific 
penalties  incurred  and  their  relative  amounts.  The  use  of  the 
zooming  technique  focuses  the  portrayal  of  only  those  years  where 
penalty  values  were  significant  to  the  model.  This  set  of  graphs 
also  demonstrates  consistency,  simplicity  and  appropriate  displays 
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f  Figure  15.  Application  of  Windowing  Environment.  This  allows  for 

efficient  on-screen  comparisons  of  additional  views  of  information 
within  the  hierarchy.  The  combination  of  bar  and  line  charts 
(front)  helps  the  analyst  quickly  spot  violations  of  constraints. 
This  is  an  example  of  a  side-by-side  comparison. 
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Figure  16.  Application  of  Windowing  Environment.  This  view 
supplements  the  on-screen  comparisons  in  Figure  15  with  the 
Production  Line  Schedules  shown  in  Figure  5  to  enable  the  analyst 
to  identify  potential  causes  of  the  Mission  Requirement  constraint 
violations  for  specific  aircraft  models.  The  side-by-side 
comparisons  and  the  user  controlled  environment  are  key  principles 
at  this  stage  of  the  analysis. 


^  Figure  17.  Windowing  Environment  with  Spreadsheet.  The  user 

control  and  data  access  principles  make  it  easy  for  the  analyst  to 
^  view  the  figures  in  the  appropriate  spreadsheet  that  support  an 

aspect  of  a  graph  under  investigation.  The  windowing  environment 
supports  the  side-by-side  comparison  of  this  information. 


79 


Figure  18.  Windowing  Environment  for  Side-By-Side  Comparisons.  The 

on-screen  combination  of  a  spreadsheet  and  two  types  of  graphs 
demonstrates  analytical  advantages.  It  highlights  the  principles 
of  user  control,  data  access  and  appropriate  displays. 
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Figure  19.  Continued  Application  o£  Windowing  Environment. 

Approaching  the  lowest  level  of  one  branch  of  the  hierarchical 
structure,  the  analyst  can  use  the  side-by-side  comparisons  to 
answer  questions  about  the  adequacy  of  the  solutions. 
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Figure  20.  Aircraft  Procurement  Data  for  Combined  Roles.  This 
graph  illustrates  the  capability  to  collect  data  from  four  roles 
into  two  combinations.  The  analysis  principle  of  extensibility 
allows  for  this  user  controlled  addition  to  the  analytical  tools  and 
backward  compatibility  would  ensure  that  this  data  change  would  be 
filtered  throughout  the  graphs  and  spreadsheets  in  all  previous 
runs . 
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Figure  21a.  Two  Case  Comparison  of  Objective  Function  Values. 

Multiple  run  analysis  is  enhanced  by  the  ability  to  create  side- 
by-side  comparisons  of  subsequent  runs  to  the  same  information  from 
a  base  case. 
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Figure  21b.  Two  Case  Comparison  of  Purchase  Cost.  Multiple  run  v 

comparisons  can  be  conducted  in  the  spreadsheet  environment  as  well 
as  graphically.  This  supports  the  analysis  principles  of  base  case 
and  side-by-side  comparisons  and,  in  conjunction  with  figure  21a, 
that  of  multiple  representations. 
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22(b) 

Figure  22.  Multiple  Run  Comparisons.  In  figure  22a  a  simple  line 
graph  serves  to  compare  total  budget  expenditures  for  a  base  case 
to  an  alternate  solution.  Figure  22b  is  a  difference  bar  chart 
supported  by  a  formula  that  subtracts  the  expenditures  for  the 
alternate  solution  from  those  of  the  base  case. 
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Figure  23.  Scatter  Plot  Comparison.  In  this  comparison  of  a  base 
case  to  an  alternate  solution  deviations  from  the  identity  (x=y) 
line  are  readily  apparent.  A  majority  of  the  points  to  the  right 
of  and  below  the  line  indicate  larger  values  for  the  base  solution 
associated  with  the  horizontal  axis. 
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Figure  25.  Multiples  Displayed  with  Area  Graphs.  The  use  of  area 
graphs  presents  the  objective  function  values  in  a  more  dramatic 
visualization  with  the  same  focus  on  similarities  and  differences. 
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Figure  26.  Boxplot.  Statistical  tools,  such  as  the  boxplot,  can 
be  used  to  compare  results  of  several  runs  of  the  model.  Results 
can  be  used  to  tighten  or  restrict  constraints  without  effecting  the 
rest  of  the  model.  The  horizontal  line  segment  inside  each  box  is 
the  50th  percentile  and  the  top  and  the  bottom  of  the  box  indicate 
the  75th  and  25th  percentiles.  The  ends  of  the  vertical  lines  are 
the  adjacent  values  which  represent  the  largest  or  smallest 
observations  within  1.5  times  the  difference  between  the  75th  and 
25th  percentile  on  either  end  of  the  box.  The  dots  above  or  below 
these  lines  represent  outlier  values. 
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